In [1]:
!mkdir -p ~/agave
%cd ~/agave
!pip3 install --upgrade setvar
import re
import os
import sys
from setvar import *
from time import sleep
# This cell enables inline plotting in the notebook
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
!auth-tokens-refresh
In [2]:
writefile("funwave-input.tpl","""
!INPUT FILE FOR FUNWAVE_TVD
! NOTE: all input parameter are capital sensitive
! --------------------TITLE-------------------------------------
! title only for log file
TITLE = VESSEL
! -------------------HOT START---------------------------------
HOT_START = F
FileNumber_HOTSTART = 1
! -------------------PARALLEL INFO-----------------------------
!
! PX,PY - processor numbers in X and Y
! NOTE: make sure consistency with mpirun -np n (px*py)
!
PX = 2
PY = 1
! --------------------DEPTH-------------------------------------
! Depth types, DEPTH_TYPE=DATA: from depth file
! DEPTH_TYPE=FLAT: idealized flat, need depth_flat
! DEPTH_TYPE=SLOPE: idealized slope,
! need slope,SLP starting point, Xslp
! and depth_flat
DEPTH_TYPE = FLAT
DEPTH_FLAT = 10.0
! -------------------PRINT---------------------------------
! PRINT*,
! result folder
RESULT_FOLDER = output/
! ------------------DIMENSION-----------------------------
! global grid dimension
Mglob = 500
Nglob = 100
! ----------------- TIME----------------------------------
! time: total computational time/ plot time / screen interval
! all in seconds
TOTAL_TIME = 4.0
PLOT_INTV = 1.0
PLOT_INTV_STATION = 50000.0
SCREEN_INTV = 1.0
HOTSTART_INTV = 360000000000.0
WAVEMAKER = INI_GAU
AMP = %AMP%
Xc = 250.0
Yc = 50.0
WID = %WID%
! -----------------GRID----------------------------------
! if use spherical grid, in decimal degrees
! cartesian grid sizes
DX = 1.0
DY = 1.0
! ----------------SHIP WAKES ----------------------------
VESSEL_FOLDER = ./
NumVessel = 2
! -----------------OUTPUT-----------------------------
ETA = T
U = T
V = T
""")
In [3]:
writefile("funwave-wrapper.txt","""
#!/bin/bash
# This is a generic wrapper script for checking the runtime environment
# of a job and verifying the runtime template variable values
set +x
date
# print out the agave runtiem variables
echo "##################################################"
echo "# Agave Job Runtime Variables "
echo "##################################################"
echo "\n"
echo 'AGAVE_JOB_NAME="\${AGAVE_JOB_NAME}"'
echo 'AGAVE_JOB_ID="\${AGAVE_JOB_ID}"'
echo 'AGAVE_JOB_APP_ID="\${AGAVE_JOB_APP_ID}"'
echo 'AGAVE_JOB_EXECUTION_SYSTEM="\${AGAVE_JOB_EXECUTION_SYSTEM}"'
echo 'AGAVE_JOB_BATCH_QUEUE="\${AGAVE_JOB_BATCH_QUEUE}"'
echo 'AGAVE_JOB_SUBMIT_TIME="\${AGAVE_JOB_SUBMIT_TIME}"'
echo 'AGAVE_JOB_ARCHIVE_SYSTEM="\${AGAVE_JOB_ARCHIVE_SYSTEM}"'
echo 'AGAVE_JOB_ARCHIVE_PATH="\${AGAVE_JOB_ARCHIVE_PATH}"'
echo 'AGAVE_JOB_NODE_COUNT="\${AGAVE_JOB_NODE_COUNT}"'
echo 'AGAVE_JOB_PROCESSORS_PER_NODE="\${AGAVE_JOB_PROCESSORS_PER_NODE}"'
echo 'AGAVE_JOB_MEMORY_PER_NODE="\${AGAVE_JOB_MEMORY_PER_NODE}"'
echo 'AGAVE_JOB_ARCHIVE_URL="\${AGAVE_JOB_ARCHIVE_URL}"'
echo 'AGAVE_JOB_OWNER="\${AGAVE_JOB_OWNER}"'
echo 'AGAVE_JOB_TENANT="\${AGAVE_JOB_TENANT}"'
echo 'AGAVE_JOB_ARCHIVE="\${AGAVE_JOB_ARCHIVE}"'
echo 'AGAVE_JOB_MAX_RUNTIME="\${AGAVE_JOB_MAX_RUNTIME}"'
echo 'AGAVE_JOB_MAX_RUNTIME_SECONDS="\${AGAVE_JOB_MAX_RUNTIME_SECONDS}"'
echo 'AGAVE_JOB_MAX_RUNTIME_MILLISECONDS="\${AGAVE_JOB_MAX_RUNTIME_MILLISECONDS}"'
echo 'AGAVE_BASE_URL="\${AGAVE_BASE_URL}"'
echo 'AGAVE_JOB_ARCHIVE="\${AGAVE_JOB_ARCHIVE}"'
echo 'AGAVE_CACHE_DIR="\${AGAVE_CACHE_DIR}"'
echo 'AGAVE_JOB_ACCESS_TOKEN="\${AGAVE_JOB_ACCESS_TOKEN}"'
echo 'AGAVE_JOB_REFRESH_TOKEN="\${AGAVE_JOB_REFRESH_TOKEN}"'
echo 'AGAVE_JOB_PACKAGE_OUTPUT="\${AGAVE_JOB_PACKAGE_OUTPUT}"'
echo 'AGAVE_JOB_COMPRESS_OUTPUT="\${AGAVE_JOB_COMPRESS_OUTPUT}"'
echo "##################################################"
echo "# Job Runtime Environment "
echo "##################################################"
echo "\n"
LD_LIBRARY_PATH=/usr/local/lib
# print environment
env
# copy to file for usage later on
env > ./environment.out
# resolve our job request parameters in funwave's input file
cp -f funwave-input.tpl input.txt
sed -i 's#%WID%#\${WID}#' input.txt
sed -i 's#%AMP%#\${AMP}#' input.txt
# run our funwave command
mpirun -np 2 /home/jovyan/FUNWAVE-TVD/src/funwave_vessel
# if the job request set the parameter compress_ouput to true
# the folder will be compressed
if [[ -z "\${compress_output}" ]]; then
tar czf output.tgz output
fi
""")
Using Agave commands, we make a directory on the storage server an deploy our wrapper file there.
In [4]:
!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N funwave-${AGAVE_APP_DEPLOYMENT_PATH}
!files-upload -F funwave-wrapper.txt -S ${AGAVE_STORAGE_SYSTEM_ID} funwave-${AGAVE_APP_DEPLOYMENT_PATH}/
!files-upload -F funwave-input.tpl -S ${AGAVE_STORAGE_SYSTEM_ID} funwave-${AGAVE_APP_DEPLOYMENT_PATH}/
All agave applications require a test file. The test file is a free form text file which allows you to specify what resources you might need to test your application.
In [5]:
writefile("funwave-test.txt","""
compress_output=1
AMP=2.0
WID=24.0
funwave-wrapper.txt
""")
In [6]:
!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N funwave-${AGAVE_APP_DEPLOYMENT_PATH}
!files-upload -F funwave-test.txt -S ${AGAVE_STORAGE_SYSTEM_ID} funwave-${AGAVE_APP_DEPLOYMENT_PATH}/
We are recycling our previous app description here with a few changes. First, we are updating the app id so a new app will be created. Second we have added three new parameters.
compress_output is a boolean parameter that tells the wrapper whether to compress the output folder after running. AMP is a numeric parameter specifying the wave amplitude. Notice the validator requires this to be a decimal value. WID is a numeric parameter specifying the width. Notice the validator requires this to be a decimal value. We have also removed the data file input from the previous app description. This is because we uploaded the input template with our app description, so it will be copied in and dynamically crated based on the runtime variable values every time the app is run.
In [7]:
writefile("funwave-app.txt","""
{
"name":"${AGAVE_USERNAME}-${MACHINE_NAME}-funwave",
"version":"1.0",
"label":"Runs a command",
"shortDescription":"Runs a command",
"longDescription":"",
"deploymentSystem":"${AGAVE_STORAGE_SYSTEM_ID}",
"deploymentPath":"funwave-${AGAVE_APP_DEPLOYMENT_PATH}",
"templatePath":"funwave-wrapper.txt",
"testPath":"funwave-test.txt",
"executionSystem":"${AGAVE_EXECUTION_SYSTEM_ID}",
"executionType":"CLI",
"parallelism":"SERIAL",
"modules":[],
"inputs":[],
"parameters":[{
"id" : "compress_output",
"value" : {
"visible":true,
"required":true,
"type":"bool",
"order":0,
"enquote":false,
"default":true
},
"details":{
"label": "Compress output folder",
"description": "If true, output will be packed and compressed",
"argument": null,
"showArgument": false,
"repeatArgument": false
},
"semantics":{
"argument": null,
"showArgument": false,
"repeatArgument": false
}
},
{
"id" : "AMP",
"value" : {
"visible":true,
"required":true,
"type":"string",
"order":0,
"enquote":false,
"default":"3.0",
"validator": "\\\\d+\\\\.\\\\d+"
},
"details":{
"label": "Wave amplitude",
"description": "Wave amplitude expressed as a decimal value",
"argument": null,
"showArgument": false,
"repeatArgument": false
},
"semantics":{
"argument": null,
"showArgument": false,
"repeatArgument": false
}
},
{
"id" : "WID",
"value" : {
"visible":true,
"required":true,
"type":"string",
"order":0,
"enquote":false,
"default":"25.0",
"validator": "\\\\d+\\\\.\\\\d+"
},
"details":{
"label": "Width",
"description": "Width expressed as a decimal value",
"argument": null,
"showArgument": false,
"repeatArgument": false
},
"semantics":{
"argument": null,
"showArgument": false,
"repeatArgument": false
}
}],
"outputs":[]
}
""")
In [8]:
!apps-addupdate -F funwave-app.txt
We have registered a new Agave app to run our funwave code. Let's update our job definition to test it out. For fun, let's post our notifications to Slack. If you do not have access to a Slack channel where you can create an incoming webhook, use the previous requestbin url from the job request file below.
In [9]:
setvar("""
EVENT=*
WEBHOOK_URL=https://hooks.slack.com/services/
WEBHOOK_URL=${REQUESTBIN_URL}?name=\\\${JOB_NAME}&event=\${EVENT}&jobid=\\\${JOB_ID}
""")
In [13]:
writefile("funwave-job.txt","""
{
"name":"funwave-1",
"appId": "${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-1.0",
"executionSystem": "${AGAVE_EXECUTION_SYSTEM_ID}",
"maxRunTime":"00:10:00",
"archive": false,
"notifications": [
{
"url":"${WEBHOOK_URL}",
"event":"*",
"persistent":"true"
}
],
"parameters": {
"compress_output":true,
"AMP":"4.0",
"WID":"20.0"
}
}
""")
Because the setvar() command can evalute $() style bash shell substitutions, we will use it to submit our job. This will capture the output of the submit command, and allow us to parse it for the JOB_ID. We'll use the JOB_ID in several subsequent steps.
In [14]:
setvar("""
# Capture the output of the job submit command
OUTPUT=$(jobs-submit -F funwave-job.txt)
# Parse out the job id from the output
JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)
""")
While the job is running, the requestbin you registered will receive webhooks from Agave every time a job event occurs. To monitor this in real time, evaluate the next cell an visit the printed url in your browser:
In [15]:
!echo ${REQUESTBIN_URL}?inspect
Of course, you can also monitor the job status by polling. Note that the notifications you receive via email and webhook are less wasteful of resources. However, we show you this for completeness.
In [16]:
for iter in range(20):
setvar("STAT=$(jobs-status $JOB_ID)")
stat = os.environ["STAT"]
sleep(5.0)
if stat == "FINISHED" or stat == "FAILED":
break
The jobs-history command provides you a record of the steps of what your job did. If your job fails for some reason, this is your best diagnostic.
In [17]:
!echo jobs-history ${JOB_ID}
!jobs-history ${JOB_ID}
This command shows you the job id's and status of the last 5 jobs you ran.
In [18]:
!jobs-list -l 5
This next command provides you with a list of all the files generated by your job. You can use it to figure out which files you want to retrieve with jobs-output-get.
In [19]:
!jobs-output-list --rich --filter=type,length,name ${JOB_ID}
Retrieve the standard output.
In [20]:
!jobs-output-get ${JOB_ID} funwave-1.out
!cat funwave-1.out
Check our resolved wrapper template to verify the variables were replaced.
In [21]:
!jobs-output-get ${JOB_ID} funwave-1.ipcexe
!cat funwave-1.ipcexe
Check the input.txt file resolved from the input template with our runtime values
In [22]:
!jobs-output-get ${JOB_ID} input.txt
!cat input.txt
Retrieve the standard error output.
In [23]:
!jobs-output-get ${JOB_ID} funwave-1.err
!cat funwave-1.err
In this next example, we run a job using an application created by the person sitting next to you. Simply edit the value for AGAVE_FRIEND
In [24]:
setvar("AGAVE_FRIEND=training001")
In [28]:
!systems-roles-list ${AGAVE_EXECUTION_SYSTEM_ID}
In [30]:
!systems-roles-addupdate -u ${AGAVE_FRIEND} -r PUBLISHER ${AGAVE_EXECUTION_SYSTEM_ID}
In [31]:
!systems-roles-list ${AGAVE_EXECUTION_SYSTEM_ID}
At this point you should be able to do the following:
In [ ]: